S&P 500 Price Uncertainty and Noise

Visualizing price trend, random variations, and forecast uncertainty

30DayChartChallenge
Data Visualization
R Programming
2025
An exploration of market uncertainty and noise in S&P 500 price data, decomposing the signal from random variations and visualizing future prediction intervals with a fan chart.
Published

April 27, 2025

Figure 1: S&P 500 price chart showing historical performance from 2022-2025, featuring a smooth trend line (signal) overlaid with actual price movements (noise). A dashed vertical line marks the present, followed by forecast uncertainty bands at 50%, 80%, and 95% confidence levels. The signal-to-noise ratio is 7.5.

Steps to Create this Graphic

1. Load Packages & Setup

Show code
## 1. LOAD PACKAGES & SETUP ----
suppressPackageStartupMessages({
pacman::p_load(
  tidyverse,      # Easily Install and Load the 'Tidyverse'
  ggtext,         # Improved Text Rendering Support for 'ggplot2'
  showtext,       # Using Fonts More Easily in R Graphs
  janitor,        # Simple Tools for Examining and Cleaning Dirty Data
  skimr,          # Compact and Flexible Summaries of Data
  scales,         # Scale Functions for Visualization
  lubridate,      # Make Dealing with Dates a Little Easier
  ggrepel,        # Automatically Position Non-Overlapping Text Labels with 'ggplot2'
  quantmod,       # Quantitative Financial Modelling Framework
  camcorder       # Record Your Plot History
  )
})

### |- figure size ----
gg_record(
    dir    = here::here("temp_plots"),
    device = "png",
    width  = 8,
    height = 8,
    units  = "in",
    dpi    = 320
)

# Source utility functions
suppressMessages(source(here::here("R/utils/fonts.R")))
source(here::here("R/utils/social_icons.R"))
source(here::here("R/utils/image_utils.R"))
source(here::here("R/themes/base_theme.R"))

2. Read in the Data

Show code
# Get S&P 500 data (just last 2 years to keep it faster)
getSymbols("^GSPC", from = "2022-01-01")
[1] "GSPC"
Show code
# Convert to data frame and prepare data
sp500_df <- GSPC |>  
  as.data.frame() |>
  rownames_to_column(var = "date") |>
  mutate(date = as.Date(date)) |>
  select(date, GSPC.Adjusted) |>
  rename(price = GSPC.Adjusted)

3. Examine the Data

Show code
glimpse(sp500_df)
skim(sp500_df)

4. Tidy Data

Show code
### |- Tidy ----
# Calculate log returns
sp500_df <- sp500_df |>
  arrange(date) |>
  mutate(log_return = c(NA, diff(log(price))))

# Decompose the signal into trend and noise components
# Use loess smoothing to extract the trend
loess_fit <- loess(price ~ as.numeric(date), data = sp500_df, span = 0.15)
sp500_df$trend <- predict(loess_fit)
sp500_df$noise <- sp500_df$price - sp500_df$trend

# Calculate noise metrics
noise_sd <- sd(sp500_df$noise, na.rm = TRUE)
noise_range <- max(sp500_df$noise, na.rm = TRUE) - min(sp500_df$noise, na.rm = TRUE)
signal_to_noise <- sd(sp500_df$trend, na.rm = TRUE) / noise_sd

# Calculate historical volatility (standard deviation of returns)
vol <- sd(sp500_df$log_return, na.rm = TRUE)

# Create forecast data
last_date <- max(sp500_df$date)
last_price <- sp500_df |> filter(date == last_date) |> pull(price)
forecast_days <- 60

# Create date sequence for weekdays only
all_dates <- seq.Date(from = last_date + days(1), 
                      by = "day", 
                      length.out = forecast_days * 1.5) # Add buffer for weekends
forecast_dates <- all_dates[!weekdays(all_dates) %in% c("Saturday", "Sunday")]
forecast_dates <- head(forecast_dates, forecast_days)

# Simple random walk simulation for forecasting
set.seed(123)
n_simulations <- 1000
simulations <- matrix(nrow = length(forecast_dates), ncol = n_simulations)

for (i in 1:n_simulations) {
  # Start with last known price
  price <- last_price
  # Generate random returns based on historical volatility
  random_returns <- rnorm(length(forecast_dates), mean = 0, sd = vol)
  
  for (j in 1:length(forecast_dates)) {
    # Apply random return
    price <- price * exp(random_returns[j])
    simulations[j, i] <- price
  }
}

# Calculate percentiles for confidence intervals
forecast_df <- data.frame(
  date = forecast_dates,
  mean = rowMeans(simulations),
  lower_95 = apply(simulations, 1, quantile, probs = 0.025),
  lower_80 = apply(simulations, 1, quantile, probs = 0.1),
  lower_50 = apply(simulations, 1, quantile, probs = 0.25),
  upper_50 = apply(simulations, 1, quantile, probs = 0.75),
  upper_80 = apply(simulations, 1, quantile, probs = 0.9),
  upper_95 = apply(simulations, 1, quantile, probs = 0.975)
)

5. Visualization Parameters

Show code
### |-  plot aesthetics ----
colors <- get_theme_colors(
  palette = c(
    "#8C1C13", "#BF4342", "#E7D7C1", "#1A4D2E"
    )
  )

### |-  titles and caption ----
# text
title_text    <- str_wrap("S&P 500 Price Uncertainty and Noise",
                          width = 70) 

subtitle_text <- str_wrap("Visualizing price trend, random variations, and forecast uncertainty",
                          width = 80)

caption_text <- create_dcc_caption(
  dcc_year = 2025,
  dcc_day = 27,
  source_text =  "Yahoo Finance via { quantmod }" 
)

### |-  fonts ----
setup_fonts()
fonts <- get_font_families()

### |-  plot theme ----

# Start with base theme
base_theme <- create_base_theme(colors)

# Add weekly-specific theme elements
weekly_theme <- extend_weekly_theme(
  base_theme,
  theme(
    
    # Text styling 
    plot.title = element_text(face = "bold", family = fonts$title, size = rel(1.14), margin = margin(b = 10)),
    plot.subtitle = element_text(family = fonts$subtitle, color = colors$text, size = rel(0.78), margin = margin(b = 20)),
    
    # Axis elements
    axis.text = element_text(color = colors$text, size = rel(0.7)),
    axis.title.y = element_text(color = colors$text, size = rel(0.8), 
                                hjust = 0.5, margin = margin(r = 10)),
    axis.title.x = element_text(color = colors$text, size = rel(0.8), 
                                hjust = 0.5, margin = margin(t = 10)),
    
    axis.line.x = element_line(color = "gray50", linewidth = .2),
    
    # Grid elements
    panel.grid.minor.x = element_blank(),
    panel.grid.minor.y = element_blank(),
    panel.grid.major.y = element_line(color = "gray65", linewidth = 0.05),
    panel.grid.major.x = element_line(color = "gray65", linewidth = 0.05),
    
    # Plot margins 
    plot.margin = margin(t = 10, r = 20, b = 10, l = 20),
  )
)

# Set theme
theme_set(weekly_theme)

6. Plot

Show code
### |-  Plot ----
p <- ggplot() +
  # Geom
  geom_line( # tend line
    data = sp500_df, aes(x = date, y = trend), 
    color = colors$palette[4], linewidth = 0.9
    ) +
  geom_line( # noise around trend
    data = sp500_df, aes(x = date, y = price), 
    color = colors$palette[4], linewidth = 0.4, alpha = 0.6
    ) +
  geom_ribbon(
    data = forecast_df,  # 95% prediction interval
    aes(x = date, ymin = lower_95, ymax = upper_95), 
    fill = colors$palette[1], alpha = 0.15
    ) +
  geom_ribbon(
    data = forecast_df,  # 80% prediction interval
    aes(x = date, ymin = lower_80, ymax = upper_80), 
    fill = colors$palette[2], alpha = 0.2
    ) +
  geom_ribbon(
    data = forecast_df,  # 50% prediction interval
    aes(x = date, ymin = lower_50, ymax = upper_50), 
    fill = colors$palette[3], alpha = 0.3,
    ) +
  geom_line(
    data = forecast_df,  # Mean forecast
    aes(x = date, y = mean), 
    color = colors$palette[1], linetype = "dashed", linewidth = 0.5
    ) +
   geom_vline(
    xintercept = as.numeric(last_date), 
    linetype = "dashed", color = "gray50"
    ) +
  # Annotation
  annotate(
    "text", x = as.Date("2022-04-01"), y = max(sp500_df$price), 
           label = "Historical Performance", hjust = 0, 
           fontface = "bold", color = colors$palette[4], size = 3.5
    ) +
  annotate(
    "text", x = as.Date("2022-04-01"), y = max(sp500_df$price) * 0.98, 
    label = paste0("Trend (signal) with noise overlay"), 
    hjust = 0, color = colors$palette[4], size = 3
    ) +
  annotate(
    "text", x = last_date + days(110), y = max(forecast_df$upper_95) * 0.95, 
    label = "95%", color = colors$palette[1], fontface = "bold", size = 3.5
    ) +
  annotate(
    "text", x = last_date + days(110), y = max(forecast_df$upper_80), 
    label = "80%", color = colors$palette[1], fontface = "bold", size = 3.5,
    vjust = 5
    ) +
  annotate(
    "text", x = last_date + days(110), y = max(forecast_df$upper_50), 
    label = "50%", color = colors$palette[1], fontface = "bold", size = 3.5,
    vjust = 5
    ) +
  annotate(
    "text", x = forecast_dates[length(forecast_dates)/4], y = min(forecast_df$lower_95) * 0.97, 
    label = paste0("Signal-to-noise ratio: ", round(signal_to_noise, 1)), 
    hjust = 0.5, fontface = "italic", size = 3
    ) +
  # Scales
  scale_y_continuous(
    labels = scales::dollar_format()
    ) +
  scale_x_date(
    date_breaks = "6 months", date_labels = "%b %Y"
    ) +
  # Labs
  labs(
    title = title_text,
    subtitle = subtitle_text,
    caption = caption_text,
    x = NULL,
    y = "Price (USD)",
  ) +
  # Theme
  theme(
    plot.title = element_text(
      size = rel(2),
      family = fonts$title,
      face = "bold",
      color = colors$title,
      margin = margin(t = 5, b = 5)
    ),
    plot.subtitle = element_text(
      size = rel(0.95),
      family = fonts$subtitle,
      color = colors$subtitle,
      lineheight = 1.1,
      margin = margin(t = 5, b = 14)
    ),
    plot.caption = element_markdown(
      size = rel(0.6),
      family = fonts$caption,
      color = colors$caption,
      lineheight = 0.65,
      hjust = 0.5,
      halign = 0.5,
      margin = margin(t = 10, b = 5)
    ),
  )

7. Save

Show code
### |-  plot image ----  
save_plot(
  p, 
  type = "30daychartchallenge", 
  year = 2025, 
  day = 27, 
  width = 8, 
  height = 8
  )

8. Session Info

R version 4.4.1 (2024-06-14 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 22631)

Matrix products: default


locale:
[1] LC_COLLATE=English_United States.utf8 
[2] LC_CTYPE=English_United States.utf8   
[3] LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.utf8    

time zone: America/New_York
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
 [1] here_1.0.1      camcorder_0.1.0 quantmod_0.4.26 TTR_0.24.4     
 [5] xts_0.14.1      zoo_1.8-12      ggrepel_0.9.6   scales_1.3.0   
 [9] skimr_2.1.5     janitor_2.2.0   showtext_0.9-7  showtextdb_3.0 
[13] sysfonts_0.8.9  ggtext_0.1.2    lubridate_1.9.3 forcats_1.0.0  
[17] stringr_1.5.1   dplyr_1.1.4     purrr_1.0.2     readr_2.1.5    
[21] tidyr_1.3.1     tibble_3.2.1    ggplot2_3.5.1   tidyverse_2.0.0

loaded via a namespace (and not attached):
 [1] gtable_0.3.6      xfun_0.49         htmlwidgets_1.6.4 lattice_0.22-6   
 [5] tzdb_0.5.0        vctrs_0.6.5       tools_4.4.0       generics_0.1.3   
 [9] curl_6.0.0        gifski_1.32.0-1   fansi_1.0.6       pacman_0.5.1     
[13] pkgconfig_2.0.3   lifecycle_1.0.4   farver_2.1.2      compiler_4.4.0   
[17] textshaping_0.4.0 munsell_0.5.1     repr_1.1.7        codetools_0.2-20 
[21] snakecase_0.11.1  htmltools_0.5.8.1 yaml_2.3.10       pillar_1.9.0     
[25] magick_2.8.5      commonmark_1.9.2  tidyselect_1.2.1  digest_0.6.37    
[29] stringi_1.8.4     labeling_0.4.3    rsvg_2.6.1        rprojroot_2.0.4  
[33] fastmap_1.2.0     grid_4.4.0        colorspace_2.1-1  cli_3.6.4        
[37] magrittr_2.0.3    base64enc_0.1-3   utf8_1.2.4        withr_3.0.2      
[41] timechange_0.3.0  rmarkdown_2.29    ragg_1.3.3        hms_1.1.3        
[45] evaluate_1.0.1    knitr_1.49        markdown_1.13     rlang_1.1.6      
[49] gridtext_0.1.5    Rcpp_1.0.13-1     glue_1.8.0        xml2_1.3.6       
[53] renv_1.0.3        svglite_2.1.3     rstudioapi_0.17.1 jsonlite_1.8.9   
[57] R6_2.5.1          systemfonts_1.1.0

9. GitHub Repository

The complete code for this analysis is available in 30dcc_2025_27.qmd.

For the full repository, click here.

10. References

  1. Data Sources:
    • Yahoo Finance via { quantmod } quantmod
Back to top